Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test clusters ( was clusterformation ) - part 3 #33264

Closed

Conversation

alpar-t
Copy link
Contributor

@alpar-t alpar-t commented Aug 30, 2018

This change focuses on the process creation and management aspect of test clusters for a single node cluster.
The goal is to be able to start using this for plugins and modules that start a single node cluster, thus the change in the plugin builder plugin to toggle it.
I was able to test this for a few actual projects. To be able to switch over to using this for plugins we will need to port the plugin install functionality.

CI changes are required so that Gradle is always ran with java > 9 and thus :buildSrc can be built with java 9 support.
This should not be a problem for external plugin authors as we already require Java 10.

Relates to #30904

- This allows to move all  all .java files from .groovy.
- Will prevent eclipse from tangling up in this setup
- make it possible to use Version from Java
```
> Task :plugins:ingest-user-agent:listElasticSearchClusters
Starting cluster: myTestCluster
   * myTestCluster: /home/alpar/work/elastic/elasticsearch/plugins/ingest-user-agent/foo
Asked to unClaimAndStop myTestCluster, since cluster still has 1 claim it will not be stopped

> Task :plugins:ingest-user-agent:testme UP-TO-DATE
Stopping myTestCluster, since no of claims is 0
```

- Meant to auto manage the clusters lifecycle
- Add integration test for cluster formation
```
> Task :plugins:ingest-user-agent:listElasticSearchClusters
Starting cluster: myTestCluster
   * myTestCluster: /home/alpar/work/elastic/elasticsearch/plugins/ingest-user-agent/foo
Asked to unClaimAndStop myTestCluster, since cluster still has 1 claim it will not be stopped

> Task :plugins:ingest-user-agent:testme UP-TO-DATE
Stopping myTestCluster, since no of claims is 0
```

- Meant to auto manage the clusters lifecycle
- Add integration test for cluster formation
Without this the tasks from different version can be ordered randomly
causing the snapshot test to fail ( i.e. creating snapshot for multiple
versions in the same location ).
Gradle happens to order them correctly when ran without "parallel" but
switches up the ordering when not.
Rather than fighting it, move to do it right before the runner
And check resulting artifact when starting up the cluster
Seems this hoggs some internal Gradle file locks, and messes with
Gradle's internals.
To be able to use process builder, the min java version for build was
bumped to 9.
This way building buildSrc will not be delayed for all invocations, and
will be less disruptive if integration tests fail
- support for multi project when not all projects apply the plugin.
    - add a test for it
- only register task liseners once per build ( not once per project )
- make the node name safe for paths and cluster name, even if it has `:` or `#` ( will be replaced by `-`)
Clean up the processes after even after `^C` is pressed.
The way Gradle does this is depending on the use of the Daemon,
but for us, even with the Daemon interupting it does not work as it
should and it eventually self terminates.

The downside is that the ES processes will be cleaned up with a delay
caused by  Gradle trying and failing to clean up the Daemon.
Allows to pass in closures that would evaluate at execution time.
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@alpar-t
Copy link
Contributor Author

alpar-t commented Sep 17, 2018

@elasticmachine test this please

Copy link
Member

@rjernst rjernst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I stared having a look at this, but the sheer size makes it difficult to review. Here are some general suggestions to start:

  • Please move unrelated changes out to separate PRs. For example, subproject -> allprojects in root build.gradle (although I don't understand why this is needed), or java 9 changes
  • Do renames in separate PRs. Moving files that show up as full delete and full add makes files almost impossible to review well since everything shows as new.
  • I have the same concerns I had in the last PR. I think all state management for the clusters should stay in the plugin, and the ElasticsearchNode should be for configuration only.

@alpar-t
Copy link
Contributor Author

alpar-t commented Oct 4, 2018

I'm closing this and opening separate PRs.

@alpar-t alpar-t closed this Oct 4, 2018
@mark-vieira mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Delivery/Build Build or test infrastructure >non-issue stalled Team:Delivery Meta label for Delivery team v6.5.0 v7.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants